DOCUMENT BESOHE 



ED 095 852 



IR 001 029 



AUTHOR 
TITLE 

INSTITUTION 

PUB DATE 
NOTE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Attala, Emile E . ; Howard, Janes A. 

Field Evaluation of a Mini Learning Resource Aided 

Instruction System. 

California Polytechnic State Univ., San Luis Obispo.; 
California Univ., Santa Barbara. 

7'4 
28p. 

HF-S0.75 HC-S1.85 PLUS POSTAGE 

♦College Students; Comparative Analysis; ♦Computer 
Assisted Instruction; Computer Programs; Conventional 
Instruction; Cost Effectiveness; Course Evaluation; 
♦Field Studies; Programing; Statistical Analysis 
BASIC; COBOL; Data General NOVA Minicomputer; 
Learning Resource Aided Instruction; Minicomputers 



ABSTRACT 

Very little work has been done in the broad field of 
computer-assisted instruction (CAI) to exploring the use of a 
minicomputer as another learning resource in the instructional 
process. Accordingly a cost-effective Learning Resource Aided 
Instruction (LRAI) System centered around a Data General NOVA 
minicomputer augmented with slide projector-audio cassette media was 
designed and developed at the University of California, Santa 
Barbara. A similar system was integrated into the educational program 
at California Polytechnic State University and experimentally 
evaluated for two courses with large enrollments. These courses were 
an introduction to COBOL and an introduction to BASIC, respectively. 
The evaluation procedure consisted of comparing the curricula of the 
two courses taught in a traditionally-aided instruction (TAI) mode 
with that taught in a LRAI mode using appropriately structured 
questionnaires, common examinations, and statistical analyses. The 
results of the evaluation strongly demonstrated that the performance 
of students using the LRAI approach was better than those in the TAI 
mode of instruction. (Author/WCH) 
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. ABSTRACT 

Very little work has been done in the broad field of Computer- 
Aided Instruction (CAI) in exploring the use of a minicomputer as 
simply another learning resource in the instructional process. 
Accordingly, a cost-effective Learning Resource Aided Instruction 
(LRAI) System centered around a Data General NOVA minicomputer 
augmented with slide projector-audio cassette media, was designed 
and developed at the University of California, Santa Barbara 
(UCSB) [1]. 

A similar system was integrated into the educational program 
at California Polytechnic State University (Cal Poly) San Luis Obi 
California and experimentally evaluated for two courses with large 
enrollments. These courses were CSc 100, an introduction to COBC:. 
and CSc 110, an introduction to BASIC, respectively. The evaluati 
procedure consisted of comparing the curricula of the two courses 
taught in a traditionally-aided instruction (TAI) mode with that 
taught in a LRAI mode using appropriately structured questionnaire 
common examinations, and statistical analyses. The results of the 
evaluation strongly demonstrated that the performance of students 
using the LRAI approach was better than the TAI mode of instructic 



I. INTRODUCTION 

Over the past ten years there has been an explosive rate of 
growth in the types and applications of minicomputers which has 
created a strong impact on the computer industry as a whole. 
Applications, that were previously in the domain of large computers 
only, are now being taken over by mini's. This trend will continue 
because of the startling progress in electronics technology which 
has driven the cost of computing equipment steadily down. Also 
in the past the minicomputer manufacturers supplied little or no 
software packages to their users; today, the most effective selling 
point of the minicomputer manufacturer is to provide the user with 
a variety of software packages to help him in his particular 
application. 

Despite the vast research endeavor in the field of computer- 
aided instruction (CAI) , little work has been done in exploring the 
use of a minicomputer as simply another learning, resource in the 
instructional process. Instead, the emphasis has been primarily 
on the use of the computer as the only available resource in a CAI 
system. Even when learning resources such as video tape and slide- 
projector audio cassette systems have been incorporated with the 
computer medium of instruction, the computer is usually allowed to 
have control over such resources via an appropriate hardware inter- 
face. One drawback of this approach is that these resources generally 
become subservient to the computer and the resulting system is still 
utilized as a CAI system rather that a Learning Resources Aided 
Instructional (LRAI) System. Furthermore, it is almost impossible 



to configure a LRAI System with off-the-shelf learning resources 
if the computer is to be in control of these resources. 

This dilemma can be resolved by configuring a LRAI system around 
a low cost minicomputer augmented with appropriate audio visual 
media. In this configuration the computer is viewed as simply 
another learning resource which the student controls in conjunction 
with the other resources in orchestrating his way, at his own pace, 
through a lesson. 

With the above viewpoints in mind a LRAI system was designed 
and developed at UCSB. A similar system was easily integrated into 
the instructional program at Cal Poly and experimentally evaluated 
for two courses with large enrollments. These courses were esc 100 
and CSc 110 introductions to programming COBOL and BASIC, respectively 
The hardware, software, courseware, and cost of these systems were 
described in detail in a paper presented at the 1973 ADCIS summer 
conference in Ann Arbor, Michigan f3].' 

This paper briefly reviews the hardware, software, and course- 
ware of the Cal Poly LRAI system. The evaluation methodologies 
and results for the CSc 100 and CSc 110 courses that were programmed 
on the system are then discussed in detail. Some general considera- 
tions regarding the cost and efficiency of a LRAI system are then 
presented. Finally some concluding remarks delineating the 
advantages of LRAI systems, their place in the educational environment 
and further research directions are outlined. 



II. CAL POLY LRAI .SYSTEM DESCRIPTION 

The LRAI system at Cal Poly is a four terminal (3 Teletypes 
and 1 CRT) system configured around a Data General 1220 minicomputer 
with 20,000 words of memory. Each terminal is augmented by a 
combination cassette tape recorder and 35 mm rear-screen slide 
projector that can easily be mounted above the terminal. Multiuser 
timesharing BASIC was chosen as the programming language because it 
was a software module available at no cost from the manufacturer, 
was easy to learn, and in general met all of the criteria established 
for a CAI language. 

Another important factor in the selection of BASIC as the LRAI 
language is that any software developed can be easily exported 
to other systems because BASIC is so widely supported, particularly 
on minicomputers. A course module on a LRAI system consists of 
a set of slides, a set of audio cassettes, and a set of BASIC 
programs. Since BASIC is exportable, a course module can easily be 
implemented on any system supporting BASIC by simply augmenting 
the system with an off-the-shelf slide projector and audio cassette. 

Several minor problems do exist with respect to exportability. 
BASIC, like all other programming languages, comes in a variety of 
dialects and is thus system dependent. Thus the BASIC programs 
representing a given course module will have to be modified to 
some extent when they are transferred to a new system. The authors 
experience is that this can be done with a minimum of effort. 

A second consideration with respect to exportability is estab- 
lishing the amount of memory required for a LRAI unit. In general, 



the minicomputer versions of BASIC which can support a LRAI system 
occupy between 6 - 8K (16-bit words) of memory. For example the 
version of BASIC employed on the Cal Poly system uses 8K of memory. 
By proper partitioning of the subject material between the various 
learning resources a LRAI system unit that contains the same material 
as a 45 minute TAI lesson can be embedded in 2K of memory. Using 
the above information an empirical relation for determining the 
memory requirements is as follows: 

TM = 2n + 8 

where 

TM = total memory needed in thousands of 16-bit words 
n = number of terminals operating simultaneously 

For example 16K of memory would be required to support four terminals 
as in the Cal Poly system. 

The first task in preparing the courseware for a L?.AI system is 
to establish the format for a lesson as depicted in the flowchart 
of Figure 1. As shown in the figure, as soon as a student signs 
on the system the program enters an initialization subroutine where, 
cliche sentences used frequently in different responses are initielized. 
Then the program goes through the introduction subroutine which 
welcomes the student to the lesson and describes the rules adopted 
for the presentation of the lesson frames to the studeiit. The 
tutoring part of the lesson then follows. Upon completion of the 
tutoring portion of the lesson the drill-and-practice session of 



the lesson is started. This part consists of a series of questions 
that test the student's comprehension of the tutoring part. At 
the termination of the drill-and-practics session, control is either 
returned to the tutoring part or the lesson is terminated. 

Preparation of the courseware for a LRAI system involves parti- 
tioning of the subject matter for a given course among the various 
learning resources comprising the system. In the case of the Cal 
Poly system this involved appropriate orchestration of the subject 
material among the slide projector, audio cassette, and minicomputer 
resources. The critical resource is the amount of memory available 
in the computer for utilization in the LRAI system. Because of this 
restriction a methodological approach was designed to distribute the 
events of a lesson among the resources of the system so that the 
equivalent of a 45 minute TAI lesson could fit into 2K of memory. 
This necessitated a highly structured approach that distributed the 
tutoring and drill-and-practice parts of the lesson on the audio- 
visual media and left in memory only the computer responses varying 
from one question to another. The strategy proved to be appropriate 
and most effective in formalizing the sequential events of a lesson 
to achieve an appropriate interaction between the student and the 
LRAI system. 



III. SYSTEM EVALUATION 

The evaluation phase of the LRAT system started in the Fall 
of 1973 at Cal Poly and consisted of comparing the system curriculum 
for COBOL and BASIC with the same curriculum taught in a TAI mode. 

BASIC is taught in two courses at Cal Poly, CSc 110 and CSc 410 
respectively. The thirteen students in the CSc 410 course were 
put on the LRAI system, while CSc 110 was taught in a TAI mode. 
Since the LRAI and the TAI samples are from two different populations 
statistical evaluation methods were not used for comparing the BASIC 
curriculum. 

The COBOL curriculum offered a more idealized environment for 
evaluation. COBOL programming attracts about 360 students per 
quarter, grouped in about 12 sections. To obtain a sample that 
truly represents this population, two sections (about 60 students) 
were randomly selected from the twelve sections. After teaching 
all sections the principles of flowcharting and some terminology 
in a TAI mode, they were then given an examination. Based on the 
results of the examination, tie two sections were divided in four 
strata and a sample of three students were randomly selected from 
each stratum. So all in all, 12 students took the CAI COBOL course 
and were grouped in four groups, Gl, G2, G3, and G4 representing 
the "D-students" , the "C-students" , the "B-students" , and the 
"A-students" respectively. 

Between the BASIC-LRAI students and the COBOL-LRAI students more 
than 400 student-hours were logged on the system during the Fall 
quarter, 1973. During this period no major delays or shut-downs 



were experienced due to system* malfunctioning. 

No conventional classes were conducted for the two LRAI groups 
Instead, the instructor was available by appointment to a student 
or a group of students to answer questions, and to facilitate the 
scheduling and operation of the LRAI room. 

♦ 

The BASIC and COBOL courseware were evaluated after each LRAI 
session and at the termination of the courses. After each session 
when the aspects which the student liked or disliked would be 
fresh in him mind, the student was asked to answer the following 
questions: 

1. What aspects of this unit did you like most, or were 
particularly effective? 

2. How could this unit be improved? Please be as specific 
as possible. 

The results of the unit evaluation were used to detect and correct 
any errors in a unit. Furthermore, the results allowed successor 
units to be modified if they had been effected by errors in prede- 
cessor units. 

The questionnaire utilized to evaluate the wiiole system at the 
end of the course is shown in Figure. 2. The letters A, B, C, D, E 
and F appearing on the form are used to facilitate the tabulation 
and plotting of the students* responses. 

Table 1 shows the responses of the 12 students of CSc 410 who 
were taught BASIC in a LRAI mode. Interesting observations which 
can be deduced from their responses are as follows: 



(a) Two students felt that LR.* T would be a detriment 
in some courses; however, they would recommend it 
to a friend. 

(b) No one was annoyed with the repetitive use of 
multiple choice questions. 

(c) Only three students were annoyed with the stereotyped 
responses they were getting from the computer. 

(d) Surprisingly enough, six students enjoyed loading 
paper tape and four more were not distracted or 

* 

annoyed by it. 

(e) Eight students enjoyed manually handling the different 

» 

learning resources. 

(f) Unanimously, the class agreed that a typical unit 
was about right in length. 

9 

Eleven out of twelve students returned their questionnaires 
trom the COBOL LRAI section. Table 2 summarizes their combined 
responses according to their group. 

Again, some interesting deductions from their responses are 
as follows: 

(a) Only one C-student felt he would have learned more from 
TAI, while two A-students felt it would have made no 
difference. 

(b) No one was annoyed with the repetitive use of multiple 
choice questions and only two C-students were annoyed with 
the stereotyped responses from the computer. 



(c) Two C-students felt annoyed with loading paper tapes. 

(d) Only one B-student felt that a typical unit was too 
long; the rest felt it was about right. 

The overall results of the questionnaire evaluation indicate 
that the LRAI approach was well accepted by both the COBOL and BASIC 
LRAI students. 

Figure 3 shows detailed plots of the grade distributions for the 
LRAI system and TAI COBOL sections. The results of the examination 
(Figure 3a) which was administered prior to selecting the LRAI 
sample shows that there are 22 TAI students separating CAI Group 
G3 and G4, 6 TAI students separating G2 from G3, and only one TAI 
student separating Gl from G2. (Recall that the LRAI COBCL sample 
was divided into four groups: Gl represented the "D-studer.ts" , 
G2 the w C-students", G3 the "B-students M , and G4 the M A-students w . ) 
A study of the homework, final examination, and total grade distri- 
butions of Figures 3b, c and d indicates the following: 

(a) All but one LRAI stduent received full credit for 
their homework, and only two TAI students could 

. achieve the same result (Figure 3b) . 

(b) On the final examination, eight out of the twelve 

CAI students scored above 80%, and only six out of the 
forty-six TAI students could achieve the same result 
(Figure 3c) . 

(c) Totaling homework, final examination, and mid-term results, 
nine out of the twelve LRAI students scored nbeve 80* , and 
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only 13 out of the 46 TAI students could achieve the 
same result (Figure 3d) . 

Similar results were achieved by the BASIC group. 
A null hypothesis test [3] was conducted only on the COBOL 
sections because the LRAI sample was selected from the same popu- 
lation. The null hypothesis that the mean score of the LRAI group 
and the mean score of the TAI group are the same was tested against 
the alternative that the mean score of the LRAI group is greater 
than the mean score of the TAI group. The following assumptions • 
were made for the null hypothesis test: 

1. The sampling came from normal populations. This assumption 
which is fairly valid in this kind of sample, is not at all 
critical when used with the t-test since the t-test is 
quite robust without this assumption. 
.2. The variances of the two populations are the same. 
The results of the null test showed that the mean score of the 
final examination for the LRAI group was significantly higher 
than the mean score for the TAI group. The level of significance 
was less than .005. The calculated value of the test statistic 
(t = 3.652) which, when compared with the percentage points of a 
t-distribution with 56 degrees of freedom, is very significant. 

In summary, the responses ot the students to the evaluation 
questionnaire demonstrated that the LRAI system is a successful 
teaching tool. Their performances during the quarter and in the 
final examination supported, without a shadow of doubt, the claim 



that when properly orchestrated and administered this node of instruc- 
tion becomes a powerful vehicle in the hands of the instructor. 

IV. LRAI COST CONSIDERATIONS 

The most fundamental principle of economics postulate that 
the only justification for any training device or medium is its 
cost effectiveness. CAI, like any other important operation or 
service, cannot be conducted successfully without a sound 
financial base. Several assumptions and attempts are available ir. 
the literature that compare the economics of CAI versus the 
economics of TAI [4]. There are also several different approaches 
along which CAI cost is presently evolving [5]. One approach 
is for a single, powerful timesharing system to serve several 
thousand terminals and cover a broad geographical region to bring 
CAI cost to the level of TAI as PLATO at the University of Illinois. 
Another approach is for a mini-system (4 to 32 terminals) to 
serve a single location thus eliminating the tremendous communi- 
cation problem the first approach will have to face. The latter 
approach, is the one taken in this research. 

It is important to point out that the authors do not advocate 
the total replacement of the TAI system of instruction by a CAI 
system. It should be clearly recognized that the LRAI system is 
designed to help the instructor and redefines his role vis-a vis 
his students. That is why it is more meaningful to postulate 
cost formulas that combine the two modes of instruction and draw 
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equal cost curves for different TAJ-LRAI combinations. Equal cos 
curves will be clearly defined from the following example, which 
was derived from true data assembled at Cal Poly. 

At Cal Poly approximately 12 sections of CSc 100 are taught 
every quarter in a TAI mode. Each section has about 30 students 
(i.e., the total number of students processed per quarter is 360) 

The total number of students (TNS) that are taught CSc 100 
at Cal Poly during a five year period is 

TNS = (5 years) (3 quarters/year) (12 sections/quarter) 
(30 students/section) 

- 5,400 students. 

Since CSc 100 is a 2 unit course (about 20 hours of TAI) the 
the total number of hours (TNH) needed is 

TNH - (20) (5400) - 108,000 hours. 

The cost of teaching 108,000 hours in a 100% TAI mode is $120,000 
This figure was derived by calculating the number of full-time 
instructors needed to teach 12 sections of CSc 100 per quarter 
(2 instructors) and multiplying their combined annual salaries 
by five.' The annual salary is based on the salary of. an Assistan 
Professor, step 3, working three quarters per year. 

Assume that a 32 terminal LRAI system (i.e., an HP 2000 F or 
PDP 11/45 time sharing systems) with a life span of five years 
existed at Cal Poly. The cost of such a system excluding termina 
but including CPU, memory, disk, 9 track magnetic tape and so for 
is in the order of $100,000. An additional $100,000 must be c.dde 



to this base cost for terminals, audio-visual equipment, and 
system maintenance and support cost for the assumed 5 year period. 
This brings the total LRAI system cost to $200,000. 

To be able to derive the cost of teaching the same number of 
students in CSc 100 in a 100% LRAI mode, the following assumptions 
and calculations are made. 

Life Span of System = (5 years) (30 weeks/year) (6 days/week) 

(12 hours/day) (32 terminals) 

« 345,600 hours 

The cost of the LRAI system per hour is 

Cost-per-hour = $ ^°'°°° - $.58 
c 345,600 

To process the same 5,400 students assume each student will need 
20 hours on the system similar to the 20 hours needed in the 
classroom, so that the. total cost will be 

■ 

5,400 x 20 x .58 - $62,640 

An equal cost curve can be derived from the relation 

%TAI ($120,000) + %LRAI (62,640) - $120,000 

Table 3 shows several solutions to the above expression for 
various percentiles of TAI and Figure 4 shows the corresponding 
equal cost curve. From the preceding discussion it is evident 
that for a LRAI system which is amortized over five years one needs 
only to reduce his manpower by 51% to achieve equal cost. It 
is also evident that after the 5 year amotization period the cost 



per contact hour on the LRAI system will become very economical. 

If efficiency is. defined as the number of students taught per 
unit cost, one can easily compare TAI efficiency versus LRAI efficienc 
as follows: 

Efficiency - Total Number of Hours 
J Total Cost 

TAI Efficiency = g^O^bOO = * 90 " ours / D °l lar 

LRAI Efficiency = = 2.59 Hours/Dollar 

which could be translated as follows: 

"Assuming equal teaching effectiveness one can process three 
times the number of students using a mini LRAI system, over 
TAI per unit cost. " 

The authors do not subscribe to this kind of philosophy. As 
has been stated several times, the teacher is an integral part of 
the LRAI system. It is mainly designed to provide him with a power- 
ful tool to use when he sees fit. What makes sense, however, is to 
reduce TAI manpower by a certain percentage and replace it with cne 
or more LRAI system to achieve an optimal system of instruction. 
Some institutions might even find a $200,000 initial investment on a 
system as the one described in the above example out of their reach. 
The advantage that the minicomputer general systems available on the 
market today offer is the possibility to start with a simplistic 
LRAI system configuration to experiment with it before expanding it 
to the $200,000 system. For example at Cal Poly the LRAI system 
initial cost was $15,000 and that included the NOVA CPU, 12K of 



memory, 2 teletypes enhanced with two audio/visual units. This 
can be easily increased to 32K of memory, 8 stations, 3 cassettes 
drives for a total cost of $40,000 prorated at about $.40/contact 
hour. This flexibility is quite an economical advantage especially 
for institutions which would l^ke to experiment and gather enough 
meaningful data before making a drastic switch in their mode of 
instruction. 

♦ 

♦ 

V. CONCLUDIN G REMARKS 

A Learning Resource Aided Instruction System utilizing a 
minicomputer with 16K of memory and slide-audio cassette media has 
been designed and evaluated. This prototype system supports 4 
stations; each user requires 2K memory and the system software 
(e.g., time-sharing BASIC) . acquires the remainder of memory. Two 
undergraduate courses with large enrollments, programming BASIC 
and COBOL respectively have been implemented on the system. These 
curricular materials have been evaluated during tne Fall of 1973 
at Cal Poly in actual COBOL and BASIC courses. 

The responses of the students to the evaluation questionnaires 
and the results of the statistical analysis performed demonstrated 
that the LRAI system is a successful teaching tool that enhances 
the role of the teacher vis-a-vis his students. The cost-analysis 
of the system demonstrated that it could be a cost-effective stand- 
alone system that even a small institution could afford and that 
the system could be easily integrated in the instructional process 



of any institution. Also the software modularity of a LRAI system 
makes its exportability easily accomplished hence it can be imple- 
mented on any system which supports BASIC. 

The LRAI system, on one hand being a general purpose system, 
can be used for more than one application. On the other hand, as 
a LRAI system, it is an ideal tool for innovative cost-effective 
educational research. The effect of adding new audio/visual 
aids to enhance teaching effectiveness can easily be investigated 
on a LRAI system. For example microfilm and/or microfiche viewers 
could be part of a LRAI station to improve comprehension of the 
material being presented. A new device called the Cue/See [6] will 
be soon tested with the LRAI system at UCSB. The advantage of the 
Cue/See device is the flexibility it gives by separating the visual 
part from the audio part on a film strip. Consequently it makes no 
difference how long the visual presentation is . discussed, only one 
frame might be needed if there is no motion involved. 

Finally, the enthusiastic reaction that this research created and 
the endorsements that mini-LRAI systems are getting in technical 
conferences strongly suggest that this kind of cost-effective 
innovative research should be encouraged and publicized. 
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Figure 1 - Flowchart for a LRAI Tutorial Unit 



LRivI questionnaire 



Th G Durnose of this survey is to help improve the LRAI system so that 
2 win better facilitate students' learning. Your honest reactions 



it will bett 
will be of great help 



1. Did you enjoy the LRAI mode of instruction? circle 1-5. 

(A) (B) (C) (D) (E) 
Enjoyed it o , didn't enjoy it 

a great deal 5 4 3 2 1 at all. 

2 Did you feel you learned more from LRAI than you probably 
would have learned from TAI over this same course materxal? 

Probably learned more Irom LRAI. 

Probably would have learned more from TAI. 

It probably wouldn't make any difference. 



(A) 
(B) 
(C) 
(D) 



I don't know, have no basis for judging the 
difference, etc. 



3. Would you take another course using LRAI if it were offered? 

(A) . LRAI would be a positive attribute to any course. 

(B) LRAI would be a positive attribute of some courses. 

(O LRAI would make no difference one way or the other. 

(D) LRAI would be a detriment in some courses. 

(E) LRAI would be a detriment in all courses. 

4 Would you recommend that a friend with interests similar to 
yours take this course in the LRAI format? 

(A) Yes 

(B) No 



(A) (B) (C) 





Annoying , distrac- 
ting/ interfered 

material. 


'Wasn't distrac-i 
ting, or didn't j 
intprfer*'-* with 1 
learning. 


Enioy^d 
it'. 


5« Repetitive use of 
multiple choice 
questions 


• 


• 




6. Stereotyped res- 
ponses of compu- 
ter to answers 








7. You load the 
paper tape 








8« You manually 

handle the differ- 
ent learning re- 
sources* (you 
cue slide/ turn 
recorder on & off , 
etc. ) 









(A) (B) (C) 



9. The typical unit was: too long about right too short 

10. Overall/ hew would you grade this course? A B C D F 



Figure 2 - Questionnaire for evaluation of LRAI syster. 

BASIC and COBOL courses. 
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Figure 3 

Grade distributions for LRAI and TAI COBOL sections. 
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Figure 3 (con't) 
Grade distributions for LRAI and TAI COBOL sections. 
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TABLE 3 
Percentiles TAI versus LRAI 



